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ABSTRACT 


In  an  earlier  paper,  the  author  has  given  some  necessary  and  sufficient 
conditions  for  the  convergence  of  iterative  methods  for  solving  the  linear 
complementarity  problem.  These  conditions  may  be  viewed  as  global  in  the 
sense  that  they  apply  to  the  methods  regardless  of  the  constant  vector  in  the 
linear  complementarity  problem.  More  precisely,  the  conditions  characterize  a 
certain  class  of  matrices  for  which  the  iterative  methods  will  converge,  in  a 
certain  sense,  to  a  solution  of  the  linear  complementarity  problem  for  all 
constant  vectors.  In  this  paper,  we  improve  on  our  previous  results  and 
establish  necessary  and  sufficient  conditions  for  the  convergence  of  iterative 
methods  for  solving  each  individual  linear  complementarity  problem  with  a 
fixed  constant  vector.  Unlike  t..e  earlier  paper,  our  present  analysis  applies 
only  to  the  symmetric  linear  complementarity  problem.  Various  applications  to 
a  strictly  convex  quadratic  program  are  also  given. 
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SIGNIFICANCE  AND  EXPLANATION 


Iterative  methods  have  been  found  very  useful  for  solving  large-scale 
linear  complementarity  and  quadratic  programming  problems.  There  are  many 
conditions  proven  to  be  sufficient  for  such  methods  to  converge.  However, 
there  are  very  few  conditions  that  are  known  to  be  necessary  for 
convergence.  Necessary  conditions  are  useful  because  they  identify  the 
underlying  limitation  of  the  methods.  In  an  earlier  paper,  we  have  been  able 
to  derive  some  necessary  and  sufficient  conditions  for  the  convergence  of  a 
large  class  of  iterative  methods.  In  the  present  paper,  we  improve  on  our 
earlier  results  and  establish  among  other  things,  the  convergence  of  many 
iterative  methods  under  the  weakest  possible  conditions. 
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MORE  RESULTS  ON  THE  CONVERGENCE  OP  ITERATIVE  METHODS 
FOR  THE  SYMMETRIC  LINEAR  COMPLEMENTARITY  PROBLEM1'2 

Jong-Shi  Pang 

1.  INTRODUCTION 

In  an  earlier  paper  (Ref.  1),  we  have  established  some  necessary  and 
sufficient  conditions  for  the  convergence  of  iterative  methods  for  solving  the 
linear  complementarity  problem.  These  conditions  may  be  viewed  as  global  in 
the  sense  that  they  apply  to  the  methods  regardless  of  the  constant  vector  in 
the  linear  complementarity  problem  (LCP) .  Specifically,  consider  the  LCP 
(q,M): 

q  +  Mx  >  0,  x  >  0  and  xT(q+Mx)  =*  0  , 
nx  n  n 

where  q  e  R  and  M  e  R  are  given  and  x  6  k  •  Let  (B,C)  be  a  Q- 

splitting  of  the  matrix  M,  i.e.  M  «  B+C  and  B  is  a  Q-matrix  (the 
LCP(q, B)  has  a  solution  of  all  vectors  q).  Let  E  be  a  nonnegative  diagonal 
matrix  with  E^  <  1.  Define  the  point-to-set  algorithmic  map  A  as 
follows:  for  all  vectors  x, 

A(x)  =*  solution  set  of  the  LCP(q+Cx,  B,  Ex) 

The  latter  LCP(r,B,s)  is  to  find  y  so  that 

r+By  >  0,  y  >  s  and  (y-s)T(r+By)  ■  0  . 

Obviously,  under  the  translation  of  variables  x  =  y-s,  the  LCP(r,B,s)  can 
be  converted  into  the  LCP(r+Bs,B).  Since  B  is  a  Q-matrix,  the  set  A(x)  is 
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nonempty  for  all  vectors  x.  Moreover,  a  vector  x*  solves  the  LCP(q,M)  if 

and  only  if  it  is  a  fixed  point  of  the  map  A,  i.e.  x*  e  A(x#). 

Given  the  diagonal  matrix  E  and  the  Q-splitting  (B,C)  of  the  matrix 

0 

M,  we  define  an  iterative  scheme  for  solving  the  LCP(q,M):  Let  x  >  0  be 

an  arbitrary  nonnegative  vector.  In  general,  given  x  >  0  (k>0),  let 

k+ 1  k 

x  be  any  vector  in  the  set  A(x  ). 

If  B  is  a  P-matrix,  then  the  set  A(x)  is  a  singleton  for  all  x.  In 
k+ 1 

this  case,  each  x  will  be  uniquely  defined. 

As  noted  in  the  earlier  papers  (Refs.  1  and  2),  the  above  fundamental 

scheme  provides  a  unifying  framework  for  the  study  of  many  well-known 

iterative  methods  for  solving  the  LCP(q,M).  In  Ref.  1,  necessary  and 

sufficient  conditions  on  the  matrix  M  have  been  given  so  that  for  all 

0  k 

vectors  q  and  all  starting  vectors  x  >0,  each  sequence  {x  }  generated 
by  the  basic  iterative  scheme  will  "converge"  to  some  solution  of  the 
LCP(q,M).  The  precise  characterization  of  convergence  is  dependent  on  the 
notion  of  convergence  involved,  on  whether  M  is  symmetric  and  on  the 
additional  property  imposed  on  the  splitting  (B,C). 

A  key  feature  of  the  analysis  in  Ref.  1  is  that  the  characterization 
applies  to  convergence  for  all  constant  vectors  q.  For  all  practical 
purposes,  it  would  be  of  interest  to  obtain  some  characterization  of 
convergence  for  each  individual  q.  A  major  objective  of  this  paper  is  to 
derive  some  necessary  and  sufficient  conditions  on  both  M  and  q  so  that 
the  basic  iterative  scheme  defined  above  will  "converge"  (to  some  solution  of 
the  LCP (q , M) )  for  all  initial  vectors  x°  >  0.  The  notion  of  convergence  is  a 
rather  delicate  one.  As  it  is  already  evident  from  the  results  in  Ref.  1,  the 
kind  of  convergence  that  one  can  characterize  depends  rather  crucially  on  the 
assumed  properties  of  the  matrix  M. 


There  are  two  main  characterizations  obtained  (Theorems  2.1  and  3.1). 

Both  require  that  the  matrix  M  be  symmetric.  The  synmetry  ot  M  seems 
rather  indispensable  for  the  kind  of  characterization  sought  after  in  this 
paper.  This  is  because  in  the  asymmetric  case,  the  analysis  employed  here 
breaks  down.  There,  a  typical  argument  for  convergence  is  contraction  of  the 
iterates  in  which  the  role  played  by  g  seems  quite  minimal.  As  a  result,  it 
might  be  necessary  to  search  for  a  different  kind  of  characterization  in  the 
asymmetric  case. 

In  addition  to  the  two  Theorems  2.1  and  3.1,  several  characterizations 
for  the  convergence  of  various  dual  iterative  methods  for  solving  a  strictly 
convex  quadratic  program  are  derived.  All  the  characterizations  are  obtained 
in  terms  of  some  very  minimal  requirements  on  the  original  problem  being 
solved.  For  example,  in  the  case  of  a  strictly  convex  quadratic  program,  the 
convergence  of  the  sequence  of  primal  vectors  induced  by  the  dual  iterative 
methods  is  characterized  by  the  feasibility  (or  equivalently,  solvability)  of 
the  primal  program  under  absolutely  no  other  conditions. 

Closely  related  to  the  LCP(q,M)  is  the  quadratic  program 

T  IT 

min  f(x)  :=  q  x  +  —  x  Mx  .  (1) 

x>0 

If  M  is  symmetric,  then  a  vector  x*  solves  the  LCP(q,M)  if  and  only  if  it 
is  a  stationary  point  of  the  above  program.  The  objective  function  f(x) 
plays  a  crucial  role  in  the  analysis  that  follows. 

We  explain  some  matrix  notations  used  in  the  paper.  If  A  is  an  n  *  m 
matrix,  a  and  8  are  subsets  of  {l,...,n}  and  (1,...,m)  respectively, 
by  AQg  we  denote  the  submatrix  of  A  whose  rows  and  columns  are  indexed  by 
a  and  8  respectively.  If  a  =  {1,...,n},  we  denote  by  A^  those  columns 
of  A  indexed  by  8.  Similar  definition  applies  to  A . 
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2.  THE  NONDEGENERATE  CASE 


We  divide  our  analysis  into  two  cases,  depending  on  whether  the  matrix 
M  is  nondegenerate  or  positive  semi-definite.  Recall  that  ~  matrix  M  is 
nondegenerate  if  all  its  principal  minors  are  nonzero.  A  well-known 
characterization  of  nondegeneracy  in  linear  complementarity  theory  is  the 
following  (Ref.  3):  M  is  nondegenerate  if  and  only  if  the  I£P(q,M)  has  a 
finite  number  of  solutions  for  all  vectors  q. 

Before  stating  our  main  result  for  the  nondegenerate  case,  we  quote  the 
following  lemma  whose  proof  can  be  found  in  Ref.  1.  The  splitting  (B,C)  of 
the  matrix  M  is  regular  if  B-C  is  positive  definite. 

Lemma  2.1.  Let  (B,C)  be  a  regular  splitting  of  the  symmetric  matrix  M. 
Then,  for  any  nonnegative  diagonal  matrix  E  with  E^  <  1  for  all  i  and 
for  any  nonnegative  vector  x, 

f(x)  -  f (y)  >  ~  { x-y)T(B-C) ( x-y)  >  0 

for  each  y  e  A(x).  Moreover,  fix)  *  f(y)  for  some  y  e  A(x)  if  and  only 
if  x  solves  the  LCP(q,M). 

The  following  theorem  is  the  main  result  if  M  is  symmetric  and  non¬ 
degenerate  . 

Theorem  2.1.  Let  M  be  a  symmetric  and  nondegenerate  matrix  and  let  q  be 

an  arbitrary  vector.  Let  (B,C)  be  a  regular  Q-splitting  of  M.  Let  E  be 

a  nonnegative  diagonal  matrix  with  E^  <  1  for  all  i.  Hien,  the  following 

three  statements  are  equivalent: 

0  k 

(A)  For  any  initial  vector  x  >  0,  any  sequence  {x  }  satisfying 

1  v 

x  e  /(x  )  is  bounded  and  thus  has  at  least  one  accumulation  point; 

moreover,  any  such  point  solves  the  LCP(q,M); 

T  IT 

(B)  The  quadratic  function  f(x)  -  q  x  +  —  x  Mx  is  bounded  below  for 
x  >  0. 

at 


n  k 

(C)  For  any  initial  vector  x°  >  0,  any  sequence  {x  }  satisfying 

k+ 1  Jr 

x  e  A( x  )  converges  to  a  solution  of  the  LCP(q,M). 

Proof .  (A)  ==>  (B).  Let  x°  >  0.  Use  the  given  x°  as  the  initial  iterate, 
generate  a  sequence  {  xk}  with  xk+  ^  e  A(xk).  By  (A),  some  subsequence 

A 

converges  to  some  solution  x  of  the  LCP(q,M).  By  Lemma  2.1,  we  have 

f(x°)  >  f(x)  . 

Since  M  is  nondegenerate ,  the  LCP(q,M)  has  a  finite  number  of  solutions. 
Consequently,  for  any  x®  >  0,  f(x®)  is  bounded  below  by  the  minimum  of  the 

A  A 

quadratic  function  values  f(x)  generated  by  a  finite  set  of  x  vectors. 

Thus  (B)  follows. 

k  Y+ 1  k 

(B)  “>  (C) .  Let  {x  }  be  any  sequence  satisfying  x  e  A(x  )  with 

x°  >  0.  According  to  the  proof  of  Theorem  4.1  in  Ref.  1,  it  suffices  to 

establish  two  things:  (i)  that  the  sequence  {x  }  is  bounded  and  (ii)  that 

the  entire  sequence  {x  }  in  fact  converges.  To  prove  (i),  suppose  that  the 

sequence  {x  }  is  unbounded.  Then  there  exist  a  nonempty  index  set  a  and  a 

ki  \  ki 

certain  subsequence  {x  }  such  that  {x^  }  +  ®  if  j  ♦  a  and  {x_.  }  is 

bounded  if  j  e  a.  Let  0  denote  the  complement  of  a. 

k 

By  Lemma  2.1,  the  sequence  {f(x  )}  is  non-increasing.  Assumption  (B) 

implies  that  {f(x  )}  is  bounded  below  and  therefore  converges.  By  Lemma  2.1 

again,  we  have 

f(xk)  -  f(xk+1)  >  j  (xk-xk+1 )T(B-C) (xk-xk+1 )  >0  all  k  . 

k  k+ 1 

Since  B-C  is  positive  definite,  it  follows  that  the  sequence  {x  -x  } 
converges  to  zero. 

ki 

Returning  to  the  subsequence  {x  },  we  have  by  definition  of  x  , 


q  +  Cx 


V1 


+  Bx 


>  0 


k 

x 


i 


k 

>  Ex 

m 


i 


-i 


(2a) 

(2b) 
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(2c) 


k  k  — 1  T  k-1  k 
(x  -Ex  )  (q+Cx  +Bx  )  *  0 

ki  k  k-1  V1 

Since  {x^  }  +  •  for  j  e  a  and  {x  -x  }  ♦  0,  it  follows  that  {x^  } 

also  tends  to  infinity  as  k^  ♦  •  for  j  8  a.  We  may  write 

k  *1”*  k  k-1 

XJ  '  E)jxJ  ‘  lx3  ~xj  >  *  <’  '  VXJ 
Since  Ejj  <  that 

ki  .  _  V1 

^  EjjXj 

for  all  k^  large  enough  and  all  j  e  a.  Hence,  by  complementarity,  we 
obtain  (cf.  (2a)  -  (2c)) 


V  \ 

0  =  (q  +  Cx  +  Bx  ) 


k-1  k,  k.  k. 

[(q  +  C(x  -  x  >)a  +  MofJxp  ]  +  M^Xp  ]  . 


(3) 


which  implies  by  the  nondegeneracy  of  M, 

k  k  “1  k  k 

i  -1  i  i  i 

a  aal  M  a  oB  B 

But  this  is  a  contradiction  because  the  left-hand  side  in  the  above  equation 

is  unbounded  whereas  the  right-hand  side  is  bounded.  Consequently,  the  entire 

k  k 

sequence  {x  }  must  be  bounded.  The  boundedness  of  {x  }  and  the  proof  of 

Theorem  4.1  in  Ref.  1  imply  the  following  conclusion:  The  sequence  {x  }  has 

at  least  one  accumulation  point  and  any  such  point  solves  the  LCP(q,M).  Since 

the  T,cP(q,M)  has  only  a  finite  number  of  solutions,  the  sequence  {x  }  has  a 


finite  number  of  accumulation  points.  Thus,  by  Ostrowski's  Theorem  which 

k 

states  that  a  bounded  sequence  {y  }  with  a  finite  number  of  accumulation 

k+ 1  k 

points  and  satisfying  ly  -y  I  ♦  0  converges  (Ref.  4,  Theorem  28.1),  the 
sequence  {x  }  indeed  converges.  This  completes  the  proof  of  (B)  “>  (C). 
(C)  “>  (A).  This  is  obvious. 
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Among  other  things/  Theorem  2.1  has  shown  that  if  M  is  symmetric  and 
nondegenerate/  and  if  assumption  (B)  holds/  then  for  any  initial  vector 
x°  >  0,  any  sequence  {xk}  satisfying  xk+1  e  A(x)  is  bounded.  This 
statement  remains  valid  if  the  nondegeneracy  assumption  of  N  is  replaced  by 
the  weaker  assumption  that  the  homogeneous  LCP(0,M)  has  zero  as  the  unique 

solution.  To  see  this,  suppose  that  lx  I  ♦  ■.  Then  the  normalized  sequence 

k  k  * 

{x  /lx  1}  has  an  accumulation  point  x  which  must  be  nonnegative  and 

^i  ^i  * 

nonzero.  Let  {x  /lx  1}  be  a  subsequence  converging  to  x.  For  each 

k^ ,  we  have 

k  -1  k.  k  -1  I*,  k 

q+Cx  +Bx  ”  q  +  C(x  -  x  )  +  Mx  >0 

ki“1  ki 

The  proof  of  (B)  ">  (C)  shows  that  the  sequence  {x  -  x  }  ♦  0.  Thus, 
ki 

dividing  by  lx  I  and  passing  the  limit  k^  ♦  •,  we  deduce  that 

A 

MX  >  0  . 

Moreover,  for  each  index  j,  we  have 


k  -1  k  k  k  -1 

0  «  (q  +  Cx  +Bx1)j(x1-Bx 

k.-l  k.  k,  k.  k  -1 

■  (q  +  C(x  -  x  )  +  Mx  ( (I-E)x  -  E(x  -  x  ))^ 


i  2 

Dividing  by  lx  I  and  noting  that  Kjj  <  1,  we  obtain  by  passing  the  limit 


‘i  ' 


x^(Mx)^  ■  0  for  all  j 


Thus  x  is  a  nonzero  solution  of  the  homogeneous  LCP(0,M).  Consequently,  if 
the  homogeneous  LCP(0,M)  has  no  nonzero  solution,  then  the  sequence  {xk} 
must  be  bounded. 
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It  is  well-known  from  quadratic  programming  theory  that  if  a  quadratic 

objective  function  is  bounded  below  on  a  feasible  region,  then  It  achieves  its 

minimum  there  (see  Refs.  5  £  6).  In  particular,  assumption  (B)  implies  that 

the  quadratic  program  (1)  has  a  solution  and  so  does  the  LCP(q,M).  Thus,  the 

existence  of  a  solution  to  the  LCP(q,M)  is  implicit  in  condition  (B). 

Moreover,  (B)  holds  if  and  only  if  the  matrix  M  is  copositive  (i.e. 

T 

x  >  0  =*=■>  x  Mx  >  0)  and  the  implication  below  holds: 

[x  >  0,  xTMx  ■  0]  =“>  qTx  >0  , 

see  Ref.  6  for  a  proof.  In  general,  the  existence  of  a  solution  to  the 
LCHq,M)  does  not  imply  (B). 

Two  splittings  that  are  particularly  interesting  are  derived  from 
Mangasarian's  iterative  procedure  (Ref.  7)  and  the  block  SOR  iterative 
method.  Mangasarian's  procedure  leads  to  a  splitting  (B,C)  of  the  form 

B  *  K  +  (Xu)-1D,  C  «  M  ~  K  -  (Xlo)-1D  and  B  -  (1-X)I  (4) 

where  X  e  (0,1],  u  >  0  and  D  is  a  positive  definite  diagonal  matrix  (see 
Ref.  1).  Specializing  Theorem  2.1  to  such  a  splitting,  we  obtain  the 
following  characterization. 

Corollary  2.1.  Let  M  be  a  symmetric  and  nondegenerate  matrix  and  q  an 
arbitrary  vector.  Let  X  e  (0,1]  and  u  >  0  be  given  scalars.  Suppose  that 
the  matrices  K  and  D  satisfy: 

(i)  D  is  a  positive  definite  diagonal  matrix 

(ii)  the  matrix  K  +  (Xu)  -  M/2  is  positive  definite 

(iii)  K  +  (Xu)  is  a  Q-matrix. 

Then  the  statements  (A),  (B)  and  (c)  in  Theorem  2.1  are  equivalent  for 
Mangasarian's  procedure. 

Proof.  It  suffices  to  observe  that  the  splitting  (B,C)  defined  in  (4)  is 
regular  and  Q  under  the  given  assumptions . 


Using  the  same  theorem  due  to  Ostrovski ,  Mangasarian  (Ref.  7)  has  proven 
the  convergence  of  the  sequence  of  vectors  generated  by  his  iterative  scheme 
under  the  assumption  that  the  matrix  M  is  symmetric,  nondegenerate  as  well 
as  copositive-plus.  Note  that  copositivity  is  not  a  pre-stated  assumption  in 
Corollary  2.1  (or  Theorem  2.1);  instead,  it  is  part  of  the  characterising 
condition  for  convergence  (implied  by  condition  (B)). 

The  block  SOR-splitting  is  defined  in  the  following  way.  Let  the 
matrix  M  be  partitioned  into  submatrices  (M^j)  with  each  diagonal  block 

being  square.  Let  D,  L  and  U  be  consisted  of  the  diagonal,  strictly 
lower  and  upper  triangular  blocks  of  M  respectively.  Then  the  block  SOR- 
splitting  (B,C)  of  M  is  given  by 

B  =  L  +  D/oi  and  C  =  U  +  (1  -  1/u>)D 

where  to  e  (0,2)  is  a  given  scalar.  See  Ref.  1.  The  point  SOR-splitting 
corresponds  to  the  case  where  each  M^j  is  the  (i,j)  entry  of  M. 

Corollary  2.2.  Let  M  be  a  symmetric,  nondegenerate  matrix  partitioned  into 
submatrices  (M^)  with  each  diagonal  submatrix  M^  being  positive 
definite.  Let  q  be  an  arbitrary  vector  and  E  any  nonnegative  diagonal 
matrix  with  E^  <  1  all  i.  Then  for  all  to  e  (0,2)  and  any  x°  >  0,  the 
sequence  {x  }  generated  by  the  block  SOR-splitting  and  the  matrix  E  is 
uniquely  defined.  Moreover,  the  statements  (A),  (B)  and  (c)  in  Theorem  2.1 
are  equivalent  for  the  block  SOR-splitting. 

Proof.  The  fact  that  the  sequence  {x  }  is  uniquely  defined  follows  from  the 
positive  definiteness  of  the  matrix  D  which  implies  that  B  is  a  P- 
matrix.  The  equivalence  of  (A),  (B)  and  (C)  follows  from  Theorem  2.1  and  the 
observation  that  the  block  SOR-splitting  Is  in  fact  regular  and  Q  under  the 
assumed  properties  (see  the  proof  of  Corollary  4.2  in  Ref.  1). 


3.  THE  POSITIVE  SEMI-DEFINITE  CASE 

The  analog  of  Theorem  2.1  for  a  positive  semi-definite  M  is  the 
following. 

Theorem  3.1.  Let  M  be  a  symmetric  positive  semi-definite  matrix  and  let 

q  be  an  arbitrary  vector.  Let  (B,C)  be  a  regular  Q-splitting  of  M.  Let 

E  be  a  nonnegative  diagonal  matrix  with  E^  <  1  for  all  i.  Then  for  any 
0  k 

initial  vector  x  >  0,  the  sequence  {x  }  generated  by  the  splitting  (B,C) 
and  the  matrix  E  is  uniquely  defined.  Moreover,  the  following  three 
statements  are  equivalent: 

0  k 

(0)  For  any  initial  vector  x  >  0,  the  sequence  {Mx  }  converges  to  some 

A  A 

vector  Mx  and  x  solves  the  LCP(q,M). 

(B)  Same  as  before. 

(E)  The  LCP(q,M)  has  a  solution. 

Before  proving  the  theorem,  we  should  point  out  that  statement  (D)  does 

k  k 

not  assert  even  the  boundedness  of  the  sequence  {x  }.  In  particular,  {x  } 

k 

could  be  unbounded.  However,  the  sequence  {Mx  }  must  converge  and  in  fact 
converges  in  a  desirable  manner. 

Among  the  equivalence  between  the  statements  (D),  (B)  and  (E)r  the  only 
non-trivial  part  that  requires  a  detailed  proof  is  the  implication  (B)  (or 
equivalently  (E))  ==>  (D).  The  equivalence  of  (B)  and  (E)  is  well-known  by 
the  symmetry  and  positive  semi-definiteness  of  M.  That  (D)  ■**>  (E )  is  also 
obvious.  Condition  (B)  (or  (E))  is  further  equivalent  to  the  feasibility  of 
the  LCP(q,M)  which  in  turn  is  equivalent  to  the  implication: 

[v  >0,  Mv  *  0]  ==>  vTq  >  0 

The  last  implication  is  satisfied  if  condition  (F)  below  holds: 

(F)  There  exists  a  vector  v  such  that  q  +  Mv  >  0. 

By  using  the  same  argument  as  in  Lemma  3  of  Ref.  7,  one  can  show  that 
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condition  (?)  (under  the  assumed  properties  of  M)  implies  that  the  sequence 
{x  }  must  be  bounded  and  thus  assertion  (D)  follows  readily.  However, 
condition  (F)  is  in  general  stronger  than  (B)  or  (E),  even  for  symmetric 
positive  semi-definite  M.  As  a  result,  the  implication  (B)  =*=>  (D)  requires 
a  separate  proof.  For  this  purpose,  we  need  a  few  lemmas.  The  first  lemma 
concerns  the  accumulation  point(s)  of  a  sequence  in  the  affine  image  of  a 
polyhedral  set.  The  second  lemma  gives  a  straightforward  property  of  a 
positive  semi-definite  matrix.  The  third  lemma  is  a  consequence  of  the 
second.  Finally,  the  fourth  lemma  concerns  the  solution  of  an  LCP  with  a 
positive  semi-definite  matrix.  Although  the  proof  of  each  of  these  lemmas  is 
fairly  easy,  we  want  to  state  the  lemmas  explicitly  because  they  all  play  an 
important  role  in  the  proof  of  Theorem  3.1. 

Lemma  3.1.  Let  S  be  a  polyhedral  set  in  Rn  and  A  any  m  x  n  matrix. 

k 

Then  any  accumulation  point  of  a  sequence  {Ax  }  where  {x  }  e  s  must  be  of 

A  A 

the  form  Ax  for  some  x  e  S. 

Proof.  This  follows  easily  from  the  observation  that  the  image  AS  is  also  a 
polyhedral  set  (see  Ref.  8)  and  is  therefore  closed. 

Lemma  3.2.  Let  C  be  an  n  *  n  symmetric  positive  definite  matrix  and  A 

T  k 

any  m  x  n  matrix.  Then  the  sequence  {ACA  y  }  converges  to  the  vector 

T*-  t  k  T™ 

ACA  y  if  and  only  if  the  sequence  {Ay}  converges  to  Ay. 

T  k  T- 

Proof .  It  is  obvious  that  if  the  sequence  {Ay}  converges  to  A  y,  then 
T  k  t- 

the  sequence  {ACA  y  }  converges  to  ACA  y.  To  prove  the  "only  if"  part, 

T  k  T* 

assume  that  the  sequence  {ACA  y  }  converges  to  ACA  y.  With  no  loss  of 

-  T  k 

generality,  we  may  take  y  =  0.  We  first  show  that  the  sequence  {Ay}  is 

T  k 

bounded.  Suppose  not.  Then  lA  y  1  +  •.  The  normalized  sequence 
T  k  T  k 

{A  y  /lA  y  1}  has  an  accumulation  point  which  must  be  nonzero  and  of  the  form 

mA 

T 

A  y  by  Lemma  3.1.  On  the  one  hand,  we  have 
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i 


T  Jr  T  k 

ACA  y  /lA  y  I  ♦  0 


because  the  numerator  tends  to  zero  and  the  denominator  tends  to  “.  On  the 

T  k  T  k  T* 

other  hand,  the  sequence  {ACA  y  /lA  y  1}  has  ACA  y  as  an  accumulation 
point.  Thus,  it  follows  that 

ACA  y  =  0 

which  implies,  by  the  positive  definiteness  of  C, 

T“ 

A  y  =  0 

T  k 

which  is  a  contradiction.  Therefore,  the  sequence  {Ay}  is  bounded. 

T  k 

Again,  by  Lemma  3.1,  any  accumulation  point  of  the  sequence  {Ay}  must  be 
T 

of  the  form  A  u  for  some  vector  u.  By  the  same  argument  just  used  above, 

t  T  k 

we  can  easily  deduce  Au  *  0.  Consequently,  the  sequence  {Ay}  converges 

to  zero  as  desired. 

T  k 

Remark .  According  to  Lemma  3.1,  the  sequence  {ACA  y  }  converges  to  some 

T- 

vector  of  the  form  ACA  y  if  and  only  if  it  in  fact  converges. 


Lemma  3.3.  Let  M  be  a  symmetric  positive  semi-definite  matrix.  Let  o  be 


any  index  set.  Then  the  sequence  {m  y  }  converges  to  the  vector  My  if 

aa  a  aa  a 

and  only  if  the  sequence  iM^y  }  converges  to  M#oy  . 

Proof.  It  suffices  to  show  the  "only  if"  part.  Suppose  that  {M^y*}  con¬ 
verges  to  Maaya.  Since  M  is  symmetric  positive  semi-definite,  we  may 
T 

write  M  =  LL  for  some  matrix  L.  Under  this  representation,  we  obtain 

M  —  L  (L  )  and  M  *  L(L  )  . 

aa  a*  a*  *a  a* 

k  T  k 

Since  M  v  3  L  (L  )  y  ,  it  follows  from  Lemma  3.2  that  the  sequence 
aora  a*  a*  J  a 

T  k  x—  k  — » 

{ (L  )  y  }  converges  to  (L  )  y  because  M  y  ♦  M  y  .  Consequently, 
a*  Ja  *  a*  Ja  aaJa  aaJa  J' 

M  y1*  =  L(L  )Tyk  ♦  L(L  )Ty  *  M  y  as  desired. 

#  ci"1  ot  oi*  •'a  ci*  "*oi  *  (a  ex 

Lemma  3.4.  Let  M  be  a  symmetric  positive  semi-definite  matrix  and  let  q  be 

1  2 

an  arbitrary  vector.  If  x  and  x  are  two  solutions  of  the  LCP(q,M), 

then  Mx  ^  =  Mx^. 
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Proof.  This  is  well-known  in  linear  complementarity  theory,  see  Ref.  9  e.g 


Proof  of  the  implication  (B)  ==>  (D)  in  Theorem  3.1.  We  first  note  that  if 
M  is  positive  semi-definite  and  (B,C)  is  a  regular  splitting  of  M,  then 
B  itself  must  be  positive  definite.  To  see  this,  let  x  ^  0  be  any 
vector .  Then 

0  <  xTMx  =  xT(B+C)x  and  xT(B-C)x  >  0  . 

Adding  the  two  inequalities  gives  xTBx  >  0.  Thus,  the  sequence  {x  }  is 
uniquely  defined. 


By  the  same  argument  used  in  the  proof  of  Theorem  2.1,  we  may  deduce  that 
k+ 1  k 

the  sequence  (x  -  x  }  converges  to  zero.  We  claim  that  the  sequence 

k  k 

{Mx  }  has  at  least  one  accumulation  point.  This  is  certainly  true  if  {x  } 

is  bounded.  Suppose  that  (x  }  is  not  bounded.  Then  there  exist  a  nonempty 

ki  ki 

index  set  a  and  a  subsequence  {x  }  so  that  {x  }  +  »  if  jea  and 

k  3 

{x^}  is  bounded  if  jea.  Let  0  be  the  complement  of  a.  As  in  the 
proof  of  Theorem  2.1,  we  may  deduce  (cf.  (3)) 

k,  k  -1  k 


'i  V 

Maaxa  *  'l{<*  +  C(x 


X  i),«  +  MaBX0i] 


for  all  k^  large  enough.  The  last  equation  shows  that  the  sequence 

k  k 

i  i 

{m  x  }  is  bounded.  By  Lemma  3.3,  the  sequence  {m  x  }  has  an 

nn  n  1  ^  •  a  a 


aa  a 

accumulation  point.  Since 


ki  ki  ki 

=  M*axa  +  M.6x0 


ki  ki 

and  {x„  }  is  bounded,  it  follows  that  the  sequence  {Mx  }  has  an 

p 


accumulation  point.  Therefore,  so  does  {Mx  }. 
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k  i 

Let  z  be  any  accumulation  point  of  the  sequence  {Mx  } .  Let  {Mx  } 


be  a  subsequence  converging  to  z.  There  exist  a  possibly  empty  index  set  a 

l.  k.  t 

and  a  subsequence  {x  }  of  {<  1}  so  that  {x.  }+*  if  j  e  a  and 

4i  3 

{Xj  }  is  bounded  if  j  e  a.  Let  6  be  the  complement  of  a.  Then  as 


before,  we  have 


*i  V1  *i  *i 

Maaxa  "  _[(q  +  C(x  ~  x  >}a  +  Ma0x0  ] 


(5) 


for  all  large  enough.  Moreover,  the  sequence  tM.axa  J  has  an 

A  A 

accumulation  point  which  must  be  of  the  form  M  x  for  some  x  >  0  by 

*0t  01  01  = 

Ai 

Lemma  3.1.  With  no  loss  of  generality,  we  may  assume  that  M  x  ♦  M  x  . 

*oioi  *a  a 
t . 

Since  the  sequence  {x-  }  is  bounded,  we  may  also  assume  with  no  loss  of 
£.  6 

generality  that  {x.  }  converges  to  some  vector  x  >  0.  It  then  follows 

0  p  = 

A  A 

that  z  =  Mx.  We  claim  that  x  solves  the  LCP(q,M).  We  have  already  noted 
that  x  >  0.  Passing  the  limit  +  •  in  (5),  we  obtain 

A 

(q  +  Mx)  =0 

1-1  *.  01 
because  x  -  x  1  *  0.  Moreover,  for  each  ,  we  have 


*.-i 

0  <  (q  +  Cx  ♦  Bx  ). 


I  -1  1  l 

i  i  i 

=(q+C(x  -x  )+Mx  ) 


6 


and 

*i  T  V  *i  *i 

(xp  >  (q  +  C( x  -  x  )  +  Mx  =  0  . 

*1  *i 

Passing  the  limit  ®  and  noting  that  x^  ♦  x^  and  Mx  ♦  Mx,  we 

conclude  that 


(q  +  Mx)^  >  0  and  *g(q  +  Mx^g  =  0 


-14- 


Consequently,  x  solves  the  LCP(q,M). 


Summarizing,  we  have  proven  that  if  z  is  any  accumulation  point  of  the 
k  * 

sequence  {Mx  },  then  there  exists  a  solution  x  of  the  LCP(q,M)  such  that 

A  ^ 

z  *  Mx.  By  Lemma  3.4,  there  is  only  one  value  for  such  Mx.  Consequently, 
k  “  ” 

the  sequence  {Mx  }  converges  to  Mx  where  x  solves  the  LCP(q,M).  This 
establishes  the  theorem. 

lc 

Remark .  Although  any  accumulation  point  of  the  sequence  {Mx  }  must  be  of 
the  form  My  for  some  vector  y  >  0,  it  is  generally  not  true  that  any 
such  y  will  automatically  solve  the  LCP(q,M).  What  is  true  is  the  existence 
of  at  least  one  such  y  which  is  a  desired  solution.  This  fact  is  also 
related  to  the  reason  why  it  is  necessary  to  follow  the  line  of  argument  used 
in  the  latter  part  of  the  above  proof. 
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4.  APPLICATIONS  TO  A  STRICTLY  CONVEX  QUADRATIC  PROGRAM. 

Theorem  3.1  can  certainly  be  specialized  to  the  two  important  splittings 
discussed  at  the  end  of  Section  2.  Instead  of  giving  these  routine  special¬ 
izations,  we  give  two  applications  of  Theorem  3.1  to  the  case  of  a  strictly 

convex  program.  Consider  the  quadratic  program 

T  IT 

minimize  c  x  +  —  x  Dx  subject  to  Ax  >  b  (6) 

where  the  matrix  D  is  symmetric  positive  definite.  The  Karush-Kuhn-Tucker 
conditions  may  be  stated  as 

T 

0  *  c  +  Dx  -  A  y  (7a) 

T 

v  =  -b  +  Ax  and  v  y  -  0  .  (7b) 

Eliminating  the  x-variables  using  (7a)  and  substituting  into  (7b),  we  obtain 
the  LCP 

v  *  -(b  +  AD  'c)  +  AD  \Ty  >  0,  y  >  0  and  vTy  *  0  (8) 

—  1  T 

where  the  matrix  AD  A  is  symmetric  positive  semi-definite.  Specializing 
Theorem  3.1  to  the  above  LCP,  we  obtain 

Corollary  4.1.  Let  D  be  a  symmetric  positive  definite  matrix.  Let  (B,C) 

-1  T 

be  any  Q-regular  splitting  of  the  matrix  AD  A  .  Let  E  be  any  nonnegative 

0 

diagonal  matrix  E  with  Eii  <  1  for  all  i.  For  any  initial  y  >  0,  the 
(uniquely  defined)  sequence  {y  }  generated  by  the  splitting  (8,0  and  the 
matrix  E  induces  a  corresponding  sequence  of  iterates  {x*}  via  (7a); 
namely 

x^  =  D  ^ ( -c  ♦  ATy*)  for  all  k  . 

The  following  two  statements  are  equivalent; 

0  k 

(D)'  For  any  y  >0,  the  induced  sequence  {x  }  converges  to  the  unique 
solution  of  the  program  (6) 

(G)  The  program  (6)  is  feasible,  or  equivalently,  solvable. 
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Proof.  We  first  observe  that  the  program  (6)  is  feasible  if  and  only  if  the 

LCP(8)  is  so.  *ttius,  according  to  Theorem  3.1,  statement  (G)  is  equivalent  to 

0  - 1  T  k 

the  fact  that  for  any  initial  y  >  0,  the  sequence  {AD  Ay}  converges  to 

-1  t“ 

some  vector  AD  Ay  and  y  solves  the  LCP(8).  To  see  that  this  latter 

lj» 

statement  is  equivalent  to  (D)1,  suppose  that  (AD  Ay)  converges  to 

—  1  T*  *  T  k 

AD  Ay  where  y  solves  the  LCP(8).  By  Lemma  3.2,  the  sequence  {Ay} 

converges  to  Ay.  Thus  the  induced  sequence  {x  }  converges  to  the  vector 

x  =  D~' (-c+ATy)  .  (9) 

It  is  obvious  that  x  is  the  unique  solution  of  the  program  (6).  Conversely, 

k  k  0 

suppose  that  the  sequence  {x  }  which  is  induced  by  {y  }  for  some  y  >  0 

converges  to  the  unique  solution  x  of  the  program  (6).  Then  there  exists  a 

vector  y  solving  the  LCP(8)  and  satisfying  (9).  It  then  follows  that  the 

T  k  T*  —i  t  k  “1  Ta 

sequence  {Ay}  converges  to  A  y.  Therefore,  AD  Ay  ♦  AD  Ay  as 

desired.  Consequently,  the  equivalence  of  (D)'  and  (G)  follows. 

A  A 

Remark.  It  is  in  general,  not  true  that  if  x  solves  the  program  (6)  and  y 

A 

is  any  vector  satisfying  (9),  then  y  solves  the  LCP(8).  Only  a  vector  of 
multipliers  y  corresponding  to  the  constraints  of  (6)  will  both  satisfy  (9) 
and  solve  ( 8 ) . 

The  significance  of  Corollary  4.1  is  the  following.  A  useful  way  to 
solve  the  quadratic  program  (6)  by  iterative  methods  is  to  apply  them  to  the 
corresponding  LCP(8).  Under  a  constraint  qualification  of  the  Slater  type, 
one  can  show  that  the  sequence  of  vectors  {y  }  generated  is  bounded  and  thus 
has  at  least  one  accumulation  point.  Moreover,  any  such  point  solves  the 
LCP(8)  and  therefore  yields  the  unique  solution  to  (6)  by  means  of  the 
relation  (9).  See  Ref.  10  e.g.  Typically  the  fact  that  any  accumulation 
point  of  the  sequence  generated  solves  the  LCP  is  an  inherent  property  of  the 
iterative  methods.  The  proof  of  the  boundedness  of  the  sequence  usually 
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requires  a  constraint  qualification  without  which  the  existence  of  at  least 
one  such  point  is  no  longer  guaranteed.  Ref.  11  contains  a  somewhat  more 
detailed  discussion  on  this  important  point.  Now,  Corollary  4.1  says  that 
under  absolutely  no  constraint  qualification  at  all,  the  very  minimal 
requirement  that  the  program  (6)  be  feasible  (or  equivalently,  solvable) 
characterizes  the  convergence  of  the  induced  sequence  {x  }  to  the  unique 
global  minimum  point  of  (6).  Although,  not  even  the  boundedness  of  the 
sequence  {y^}  is  asserted,  the  convergence  of  the  induced  sequence  {x  } 
should  have  served  all  desired  practical  purposes. 

In  a  similar  way,  the  significance  of  Theorem  3.1  is  also  easily  seen. 
Indeed,  as  we  have  mentioned  earlier,  condition  (F)  which  is  a  Slater 
constraint  qualification  implies  conclusion  (D).  Theorem  3.1  says  that  the 
very  minimal  requirement  that  the  LCP(q,M)  be  feasible  (or  equivalently, 
solvable)  characterizes  the  convergence  of  the  sequence  of  vectors 

k  k  a  ^ 

{w  =  q+Mx  }  to  a  desired  solution  w  =  q+Mx  of  the  LCP(q,M).  Again,  the 

characterization  requires  no  such  constraint  qualification  as  (F). 

There  is  yet  another  approach  to  apply  an  iterative  method  for  solving 
the  strictly  convex  quadratic  program  (6).  This  approach  was  first  proposed 
by  Han  and  Mangasarian  (Ref.  12)  as  a  special  case  of  their  exact  penalty 
function  theory  for  general  nonlinear  programs.  In  what  follows,  we  describe 
this  approach  from  a  different  point  of  view. 

The  prime  motivation  for  this  alternative  formulation  has  to  do  with  the 
fact  that  the  LCP(8)  involves  the  inverse  of  the  matrix  D.  For  large-scale 
applications  with  sparse  data,  it  is  not  very  desirable  to  invert  D  because 
the  inversion  can  easily  destroy  the  sparsity  structure.  As  a  result,  one  is 
led  to  investigate  the  formulation  (7)  from  which  (8)  was  derived.  However, 
(7)  is  defined  by  the  matrix 
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which  contains  a  zero  diagonal  block.  This  zero  diagonal  block  prohibits  the 
application  of  say,  the  point  SOR-method.  As  a  remedy  for  this,  one  observes 
that  (7)  is  equivalent  to 


T 

0  =  c  +  Dx-Ay 

v  =  -b  +  Ax  -  YA{c+Dx-ATy)  >  0,  y  >  0  and  vTy  »  0 


which  can  be  rewritten  as 

0  =  c  +  Dx  -  ATy  (7a) 

v  =  -b  -  YAc  -  A(YD-I)x  +  YAATy  >  0,  y  >  0  and  vTy  *  0  (7b)' 

where  Y  is  some  positive  scalar.  If  A  has  no  vanishing  rows,  the  matrix 
T 

YAA  has  positive  diagonal  entries.  Thus  the  point  SOR-method  is 
applicable.  However,  the  matrix 


is  nonsymmetric  and  most  likely,  not  even  positive  semi-definite.  One  way  to 
symmetrize  the  above  matrix  is  to  multiply  the  expression  (7a)  by  the  matrix 
YD-I  which  will  be  nonsingular  (in  fact,  positive  definite)  if  Y  >  VP 
where  p  is  the  least  eigenvalue  of  the  positive  definite  matrix  D.  Thus, 
if  Y  >  1/P,  problem  (7)  is  equivalent  to 

0  “  (YD-I)c  +  (YD-I)Dx  -  (YD-I)ATy  (10a) 

T  T 

v  =  -b  -  YAc  -  A(YD-I)x  +  YAA  y,  y  >  0  and  v  y  ■  0  .  (10b) 

This  last  formulation  (10)  is  precisely  the  one  to  which  Han  and  Mangasarian 
(Ref.  12)  proposed  the  application  of  the  point  SOR-method.  They  have  shown 
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that  for  Y  >  1/P,  the  matrix 


((YD-I)D  -(YD-I)AT  \ 

]  (11) 

-A(YD-I)  YAAT  / 

is  symmetric  positive  semi-definite  and  that  if  Y  >  1/P,  the  problem  (10) 
has  a  solution  (x(Y),  y(Y))  such  that  x(Y)  ■  x*  where  x*  is  the  unique 
solution  of  the  program  (6),  provided  that  the  program  (6)  is  feasible.  This 
latter  conclusion  also  follows  easily  from  our  derivation  of  the  problem 
(10).  He  remark  that  if  Y  >  1/P  and  A  has  linearly  independent  rows,  then 
the  matrix  M  in  (11)  is  positive  definite. 

Concerning  the  convergence  of  the  point  SOR-method,  Ban  and  Mangasarian 
(Ref.  12)  have  shown  that  if  either  (i)  the  matrix  A  has  no  vanishing  rows 

A 

and  has  linearly  independent  columns,  and  there  exists  a  vector  x  with 

A 

Ax  >  b,  or  (ii)  the  matrix  A  has  linearly  independent  rows,  then  for 

Y  >  1/P,  for  any  relaxation  parameter  u  e  (0,2)  and  any  initial  vector 

(x°,y°)  with  x°  arbitrary  and  y°  >  0,  the  sequence  (x*,y*)  generated  is 

bounded  and  thus  has  at  least  one  accumulation  point.  Moreover,  any  such 

point  (x,y)  solves  (10).  In  fact,  x  is  the  unique  optimum  solution  of  (6) 

to  which  the  sequence  {x  }  must  converge. 

The  above  convergence  result  of  Han  and  Mangasarian  ns  proven  under  seme 

linear  independence  property  of  the  matrix  A  and  in  at  least  one  instance, 

under  a  constraint  qualification  as  well.  The  corollary  below  shows  that 

under  absolutely  no  such  restriction  on  A  and  no  constraint  qualification, 

the  same  convergence  of  the  sequence  {x*}  can  be  asserted.  However,  as  in 

the  two  previous  results,  (Theorem  3.1  and  Corollary  4.1),  only  the 

T  k  k 

convergence  of  the  sequence  (Ay),  and  not  the  boundedness  of  {y  },  can 
be  proved.  Again,  for  all  practical  purposes,  it  is  the  convergence  of  the 
sequence  {x  }  that  is  of  interest. 
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Corollary  4.2.  Let  D  be  a  symmetric  positive  definite  matrix  with  least 
eigenvalue  p  >  0.  Let  A  be  any  matrix  with  no  vanishing  rows.  Fix 

Y  >  1/p.  Then  for  any  w  e  (0,2)  and  any  initial  vector  (x°,y°)  with  x° 

A  1c  )c 

arbitrary  and  y  >  0,  the  sequence  of  iterates  {(x  ,y  )}  generated  by  the 

point  SOR-method  is  uniquely  defined.  Moreover,  the  following  two  statements 

are  equivalent: 

(H)  For  any  w  e  (0,2)  and  any  initial  vector  (x°,y®)  with  x°  arbitrary 

0  k  k  k 

and  y  >  0,  the  sequence  {(x  ,y  )}  is  such  that  {x  }  converges  to  the 

T  k 

unique  solution  x*  of  the  program  (6)  and  that  {Ay}  converges  to  some 
vector  ATy*  where  (x*,y*)  solves  the  problem  (10); 

(G)  same  as  before. 

Proof.  Although  the  formulation  (10)  is  not  exactly  the  one  of  a  standard 
LCP,  the  same  analysis  of  Theorem  3. 1  allows  one  to  conclude  the  equivalence 
of  the  following  two  statements; 

(G)  '  The  problem  (10)  has  a  solution. 

(H) '  For  any  we  (0,2)  and  any  initial  vector  (x°,y°)  with  x°  arbitrary 

0  k  v 

and  y  >0,  the  sequence  {Mz  }  where  M  is  given  by  (11)  and  zK  * 

k  k 

( x  #y  )  is  the  sequence  generated  by  the  point  SOR-method ,  converges  to  some 

A  A  A  A  A 

vector  Mz,  where  z  •  (x,y),  and  z  solves  (10). 

Noting  that  the  matrix  YD“I  is  nonsingular  if  y  >  1/p,  we  deduce  that 
problem  (10)  has  a  solution  if  and  only  if  (7),  or  equivalently  (6),  has 
one.  Consequently,  statements  (G)  and  (G)'  are  equivalent.  Noting  that 


k  t  k 

(YD-I)(Dx*  -  Ay) 


k  k  T  k 

Ax  +  YA(Dx  -  Ay) 


we  may  easily  deduce  that  (H)  “>  (H)’.  Conversely,  suppose  that  Mz  +  Mz 

A  A  A 

and  z  -  (x,y)  solves  (10).  Since  YD-I  is  nonsingular,  it  follows  from  the 
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expression  (12)  that 


k  T  k  T*  k  A 

Dx  -Ay  +  Dx  -  A  y  and  Ax  ♦  Ax 


(13) 


lc 

We  claim  that  the  sequence  {x  }  is  bounded.  Suppose  not.  Then  lx  I  +  ■ 

)c  1c 

and  the  normalized  sequence  {x  /lx  1}  has  a  nonzero  accumulation  point  u. 

With  no  loss  of  generality,  we  may  assume  that  u  is  the  limit  of  the  entire 
k  1c 

sequence  {x  /lx  I}.  It  follows  from  (13)  that 

(Dx*  -  ATy*)/lx*l  ♦  0  and  Au  =  0  . 

k  )c  T  k  k 

Since  {Dx  /lx  1}  is  itself  converging  to  Du,  the  sequence  {A  y  /lx  1} 

T 

must  converge  and  by  Lemma  3.1,  it  must  converge  to  A  v  for  some  vector 
v.  Thus,  we  have 

m 

0  . 


T 

Du  -  A  V 


Pre-multiplying  the  above  identity  by  uA  and  using  the  fact  that  Au  ■  0# 

we  deduce  u  Du  =  0  which  is  impossible  because  D  is  positive  definite 

k  ^ 

and  u  is  nonzero.  Consequently,  the  sequence  {x  }  is  bounded.  Let  x  be 

k  -  A  *1 

an  accumulation  point  of  {x  } .  We  claim  that  x  =  x.  Indeed,  let  {x  }  be 

k  T  k 

a  subsequence  converging  to  x.  Since  the  sequence  {Dx  -A  y  }  is 

T  *i 

convergent,  it  follows  that  the  subsequence  {Ay  }  must  itself  be 

T 

converging  and  by  Lemma  3.1,  to  a  vector  of  the  form  A  v.  Thus,  we  have 

Aw  T  *  a,  A 

Dx-Av-Dx-Ay  and  Ax  «  Ax 


or  equivalently. 


/  x  -  x 


v  -  y 


which  implies,  by  the  positive  definiteness  of  D, 

A 

Ar 

X  *  X 

k  A  T  k 

Consequently,  the  sequence  {x  }  converges  to  x  and  the  sequence  {Ay} 

fpA  A  A  A 

converges  to  A  y.  Since  (x,y)  solves  (10),  it  follows  that  x  is  the 
unique  solution  of  the  quadratic  program  (6).  Therefore,  assertion  (H) 
follows.  This  completes  the  proof  of  the  corollary. 
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Remark.  At  the  point  where  we  have  established  the  boundedness  of  the 
sequence  {x  },  we  cannot  use  the  Kan  and  Manga sari an  convergence  result  to 
conclude  that  (x  }  converges  to  the  unique  solution  of  (6).  This  is  because 
for  the  Kan  and  Mangasarian  result  to  be  applicable,  we  need  to  know  the 
boundedness  of  the  sequence  {y  }  as  well.  However,  the  boundedness  of 
{y  }  is  generally  not  guaranteed  without  the  linear  independence  property  of 
the  matrix  A. 


5.  ON  LAGRANGIAN  RELAXATION 

In  a  recent  report  (Ref.  13} ,  Cottle  and  Duvall  have  described  a 

Lagrangian  relaxation  algorithm  for  a  constrained  matrix  problem  which  Is 

formulated  as  a  strictly  convex  separable  quadratic  program.  They  proved  the 

convergence  of  their  algorithm  under  a  strong  consistency  condition  which  is  a 

constraint  qualification  of  the  Slater  type.  In  what  follows,  we  use  the 

previous  analysis  to  derive  a  convergence  result  for  the  Lagrangian  relaxation 

method  under  no  such  constraint  qualification.  As  a  matter  of  fact,  we  shall 

treat  the  following  more  general  strictly  convex  quadratic  program 

IT  T 

minimize  -  x  D*  +  c  *  subject  to  Ax  >  b  and  x  e  X  (14) 

where  the  matrix  D  is  symmetric  positive  definite  and  the  set  X  is 
polyhedral : 

X  -  {x  e  Rn  s  Fx  >  f}  . 

As  with  the  previous  problem  (6),  we  have  included  only  inequality  constraints 

in  the  program  (14).  This  is  done  for  the  sake  of  consistency;  the  same 

analysis  is  equally  applicable  to  programs  with  both  inequality  and  equality 

constraints  (such  as  the  constrained  matrix  problem  in  Ref.  13). 

The  Lagrangian  dual  function  of  the  program  (14)  is 

d(y)  -  min  xTDx  +  cTx  +  yT(b  -  A^x)  (15a) 

xex 

and  the  dual  of  (14)  can  be  defined  as 

max  d(y)  .  (15b) 

y>0 

The  Lagrangian  relaxation  approach  for  solving  the  program  (14)  may  be 
described  as  follows.  Let 
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0  o  0 

be  a  partition  of  the  rows  of  the  matrix  A.  Let  y  ■  (y  )  be  an 

arbitrary  nonnegative  vector  partitioned  in  accordance  with  A.  In  general, 

let  yK  *  (y1,...«y1|)  >0  be  given.  To  obtain  yK  =  (y1  ,...,yN  ), 

solve  N  quadratic  subprograms  consecutively  for  a  »  1,...,N: 

.  ,  1  T  .  .  T  r  ,  k+1,T.  Y  ,  k.T  . 

minimize  -  x  Dx  +  (c  -  2  (yfl  )  A-  -  I  (y.)  A  )x 

2  »«  6  6  *>«  6  6  (16) 

a 

subject  to  A  x  >  b  and  x  e  X 

a  =  a 

k-H 

and  let  y^  >0  denote  the  vector  of  Lagrange  multipliers  for  the 

constraints  (A  x  >  b). 

a  *  a 

In  order  for  the  above  Lagrangian  relaxation  method  to  be  practically 

useful,  it  is  important  that  the  partitioning  of  the  matrix  A  should  be  such 

that  each  subproblem  (16)^  is  very  easy  to  solve.  This  is  indeed  the  case  for 

the  constrained  matrix  problem,  see  Ref.  13. 

As  noted  by  Cottle  and  Duvall  (Ref.  13),  the  above  Lagrangian  relaxation 

algorithm  can  be  interpreted  as  a  block  cyclic  ascent  method  applied  to  the 

k+ 1 

dual  program  (15).  Indeed,  it  is  easy  to  see  that  the  vector  y^  solves 
the  dual  subprogram 

max  d({yj+,]6<a  ,  yQ,  {yj}0>a) 

v° 

which  is  obtained  from  (15b)  by  fixing  all  variables  except  those  of  y^. 

In  order  to  state  our  convergence  result  for  the  Lagrangian  relaxation 
method,  we  give  a  different  interpretation  of  the  method  in  the  context  of 
matrix  splittings.  To  simplify  the  notations,  we  take  N  =  2  and  consider 
the  matrix  A  partitioned  into  two  sets  of  rows  A1  and  Aj.  Then,  using 
the  system  of  linear  inequalities  which  define  the  set  X,  we  may  state  the 
Karush-Kuhn-Tucker  conditions  of  the  program  (14)  as: 
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0  = 

c 

+  Dx 

.T 

-  A1y1 

.T  „T 

-  v2  -  F  2 

X 

unrestricted 

(17a) 

W1 = 

-b1 

+  A  ,jX 

>  0 

t 

yl 

> 

0 

T 

'  Wlyl  = 

0 

(17b) 

w2  = 

"b2 

+  A2x 

>  0 

f 

y2 

> 

0 

T 

'  W2y2  * 

0 

(17c) 

V  = 

-f 

+  Fx 

>  0 

9 

z 

> 

0 

T 

,  v  z  = 

0  . 

( 17d) 

Solving 

for 

x  in 

(17a) 

and  substituting  into 

(17b) 

-  (17d),  we 

obtain 

an 

LCP(q,M)  where 


(h'] 

l*A 

-1 

!A'\ 

q  =  - 

(N 

£> 

A2 

D  c  and  M  = 

A2 

\<l 

\  F  ) 

lF 

-1  T  T  T 
D  '(Aj  A*  F1) 


(18) 


iterate  y 


k+1 


=  (y 


k+1  k+1 

■y2 


lt\ 


W 


Bk  - 


k  .  k 

given  y  =  (y^y 

2>  >0  and  zk  >  0, 

the  next 

)  produced  by  the 

Lagrangian  relaxation 

algorithm 

following  ways 

uk ,  Bk )  where 

> 

a 

i 

o 

*1D"V  \ 

A2D_1A^  a2d-1a- 

1 

,  „-1  T 

a2d  f 

1 

Ck  =  M  - 

Bk 

\fd_1a^  0 

fd_1ft  1 

and  let 


*4 


/  k+1 

/yl 


-1 


be  a  solution; 
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i 


k+  2  k+  2  k+  2 

(ii)  Solve  the  LCP(q  +  C  u  ,  B  )  where 


/-IT  -IT  -1  t\ 
/aid  a,  a,d  a2  a,d  f  \ 


B 


V 


-IT  -IT 

o  a2d  a2  a2d  P 

-IT  -IT  / 
0  FD  A2  FD  F  / 


and  let 


k+i 


M  -  B 


k+  2 


k+1 


rn 


k+1 


VW 


be  a  solution. 


-1 


Observe  that  in  step  (i)  above,  the  subvector  y2  is  actually  useless 
for  all  subsequent  computations,  thus  it  can  be  ignored.  Moreover, 


k+  ! 

V+1  *  2  ~k  ~k 

(y^  ,z  )  is  a  solution  to  the  LCP(q  ,B  )  where 


~k 

q 


'1\  /"l’ 


-1,  .T  k.  .  ~k 
D  (c  -  A2y2>  and  B 


- 1  T  T 
ID  ’(A^Fj) 


The  latter  LCP  is  clearly  equivalent  to  the  subprogram  ( 16  >  1 .  Similarly,  step 
(ii)  is  equivalent  to  the  subprogram  ( 16 ) 2  and  the  subvector  yk+1  in  uk+1 
can  be  ignored  in  the  computations. 
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To  summarize  the  above  discussion,  we  conclude  that  the  Lagrangian 


relaxation  algorithm  can  be  interpreted  as  a  special  realization  of  our  basic 
iterative  scheme  for  solving  the  L CP  in  which  the  splitting  (B*,C*)  is 
changing  from  one  iteration  to  another  in  a  cyclic  order.  With  this 
interpretation  on  hand,  we  can  now  state  and  prove  our  convergence  result  for 
the  Lagrangian  method  under  no  constraint  qualification  on  the  original 
program  (14). 

Theorem  5.1.  Let  D  be  a  symmetric  positive  definite  matrix.  Suppose  that 

each  of  the  subprograms  (16)^  is  feasible.  Let  x(y  '  )  denote  the  unique 

optimum  solution  of  (16)^  where 

'k,a  ,  k+1  k+1  k  k. 

y  =  (y1  . ya-1 '  ya . V  * 

Then  the  following  two  statements  are  equivalent: 

00  ~  *k,a  , 

(I)  For  any  y  =  (ya)  >  0,  the  sequence  lx(y  )}  converges  to  the  unique 
global  minimizer  of  the  program  ( 14 ) ; 

(J)  The  program  (14)  is  feasible,  or  equivalently,  solvable. 

Proof.  It  suffices  to  show  (J)  =«*>  (I).  Suppose  that  (14)  has  an  optimum 

solution  x  which  must  necessarily  be  unique.  Let  {x(y*'a)}  be  the 

0  0 

sequence  generated  by  some  initial  vector  y  *  (y^)  =  0.  We  sketch  the  proof 
~  *k  a  * 

that  {x(y  '  )}  +  x  for  the  case  N  ■  2.  The  argiment  for  an  arbitrary  N 

is  similar.  According  to  the  matrix-splitting  interpretation  of  the 

Lagrangian  relaxation  method,  we  see  that  the  vector  y  is  obtained  from 

.  1  .  1  1 

v  .  .  .  *+  T  T  k+  - 

yK  by  solving  two  LCP’s,  namely  (q  +  CKu  ,BK)  and  (q  +  C  u  ,B  *). 

The  proof  that  {x(y  '  )}  converges  to  x  is  essentially  the  same  as  that 

of  Corollary  4.1  which  is  a  consequence  of  Theorem  3.1.  However,  a  key 

requirement,  namely,  that  {u  -  u  }  ♦  0  in  the  proof  of  the  implication 


(B)  »*»>  (D)  in  Theorem  3.1  is  not  guaranteed  in  the  present  situation.  This 

k  k  k+j 

is  because  the  splittings  (B  ,C  )  and  (B  ,C  )  are  not  necessarily 


)r  Jr 

regular.  In  fact,  it  can  easily  be  shown  that  the  matrices  B  -  C  and 

k+  ~  k+  j 
2  2 

B  -  C  are  positive  semi-definite  but  need  not  be  positive  definite. 

A  closer  look  at  the  proof  of  (B)  •*=>  (D)  in  Theorem  3.1  reveals  that  if 


cC+  * 


-  uk)  +  0  and  C  2(uk+1  -  u  2)  ♦  0  , 


(19) 


then  the  conclusion  that  the  sequence  {Mu  }  (and  also  the  sequence 
k+  ± 

(mu  ^})  converges  to  some  Mu  where  u  solves  the  LCP(q,M)  remains 

valid,  thus,  if  (19)  holds,  the  convergence  of  {x(yk'a)}  to  x  follows  as 

in  Corollary  4.1.  The  proof  that  (19)  holds  resembles  that  of 
k+1  k 

(x  -  x  }  +  0  in  Theorem  3.1  and  is  not  repeated  here. 

Toward  the  end  of  the  report  (Ref.  13),  Cottle  and  Duvall  presented  some 

computational  experience  to  support  the  superiority  of  a  modified  version  of 

the  above  Lagrangian  method.  This  modified  version  involves  the  relaxation  of 
k+1 

each  iterate  y^  as  in  a  block  SOR  method.  At  present  time,  we  are  not 

able  to  prove  the  convergence  of  this  modified  method.  The  main  difficulty  is 

that  we  can  not  prove  the  key  descent  property  (cf.  Lemma  2.1)  when  the 
k+1 

iterates  ya  are  overrelaxed.  This  difficulty  can  be  partially  explained 

by  considering  the  matrix- splitting  interpretation  of  the  Lagrangian  method. 

k+1  k+1 

Consider  the  case  N  *  2.  The  vectors  y1  and  y2  appear  as  subvectors 

~2  y+ 1  k+1  k+2 

in  u  and  u  .  By  overrelaxing  only  y1  and  y2  ,  and  not  the 

k+  2  k+1 

entire  vectors  u  and  u  ,  it  is  not  clear  whether  the  quadratic 


function 


T  IT 

9(u)=qu+— uMu 


where  q  and  M  are  given  in  (18)  will  have  the  desired  descent  property. 
After  all,  this  descent  property  has  played  such  an  indispensable  role  in  all 


the  convergence  results  obtained  here 


6.  ON  GRADIENT-TYPE  METHODS 

In  the  previous  two  sections,  we  have  studied  the  convergence  of  three 
iterative  approaches  for  solving  a  strictly  convex  quadratic  program.  All 
three  approaches  are  based  on  a  dual  formulation  of  the  original  program. 

There  is  a  fourth  approach  which  is  also  based  on  a  dual  formulation  and  which 
makes  use  of  a  gradient-type  method  for  unconstrained  optimization.  This 
latter  approach  was  proposed  by  Ha  {Ref.  14)  for  solving  structured,  largr- 
scale  quadratic  programs .  To  describe  Ha's  approach,  consider  the  quadratic 
program 

1  T  T 

minimize  —  x  Dx  +  c  x  subject  to  Ax  =  b  and  x  @  X  (20) 

where  D  is  a  symmetric  positive  definite  matrix  and  X  is  a  polyhedral 
set.  (In  the  case  where  D  is  positive  semi-definite,  Ha  has  proposed  the 
use  of  the  proximal  point  algorithm  (Ref.  15)  to  strongly  convexify  the 
objective  function.  Much  more  detailed  discussions  on  how  the  proximal  point 
algorithm  can  be  used  as  a  decomposition  method  for  solving  large-scale 
structured  convex  programs  can  be  found  in  Ha's  Ph.D.  dissertation  (Ref. 

16).)  The  dual  of  (20)  may  be  stated  as 

maximize  d(y)  :  y  unrestricted  (21) 

where  d(y)  is  as  given  by  (15a).  If  the  set  X  is  nonempty,  then  the  dual 
function  is  finite,  concave  and  differentiable  for  all  y.  However,  d(y)  is 
typically  not  strongly  concave.  The  gradient  of  d(y)  is  given  by 

Vd(y)  =  b  -  Ax ( y )  (22) 

where  x(y)  is  the  unique  global  minimizer  of  the  Lagrangian  function 

IT  T  T 

L( x , y )  =  —  x  the  +  c  x  +  y  (b-Ax) 

over  x  e  X. 
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Problem  (21)  is  an  unconstrained  maximization  program  with  a  concave 
everywhere  differentiable  objective  function.  (The  unrestrictedness  of  the  y- 
variables  is  due  to  the  equality  constraints  Ax  *  b.  In  general,  given  a 
strictly  convex  quadratic  program  such  as  (14)  where  there  are  inequality 
constraints,  the  process  of  converting  such  inequality  constraints  to  equality 
constraints  by  adding  or  subtracting  slack  variables  will  destroy  the  strict 
(equivalent  to  strong  in  the  quadratic  case)  convexity  of  the  objective 
function;  thus  a  strong  convexif ication  procedure  might  be  necessary  in  such  a 
case.)  Ha  (Ref.  14)  proposed  the  use  of  a  gradient-type  method  for 
unconstrained  optimization  to  solve  (21).  A  point  of  concern  here  is  the 
boundedness  of  the  sequence  generated.  Typically,  such  a  boundedness 
conclusion  follows  as  an  immediate  consequence  of  a  boundedness  assixnption  on 
the  level  sets  of  the  function  to  be  optimized.  In  the  case  of  problem  (21), 
we  have  the  following  characterization. 

Proposition  6.1.  Let  D  be  a  symmetric  positive  definite  matrix  and  let  X 
be  a  nonempty  polyhedral  set.  Then  the  dual  function  d(y)  defined  in  (15a) 
has  bounded  level  sets  if  and  only  if  there  exists  a  neighborhood  U  of  b 
such  that  the  system 

Ax  -  b’ 

has  a  solution  in  X  for  all  b'  e  U.  In  particular,  if  d(y)  has  bounded 
level  sets,  then  A  must  have  linearly  independent  rows. 

Proof.  According  to  Corollary  14.2.2  in  Ref.  8,  the  function  d(y)  has 

•  • 

bounded  level  sets  if  and  only  if  0  6  int(dom  d  )  where  dom  d  denotes  the 
effective  domain  of  the  conjugate  function  d  of  d.  By  an  easy 
manipulation,  it  can  be  shown  that 

dom  d  =  {z  :  the  system  Ax=b-z,  x6X  is  consistent) 

Thus  0  e  int(dom  d  )  if  and  only  if  there  exists  a  neighborhood  N  of  the 


-31- 


origin  such  that  for  all  vectors  z  e  N  the  system  Ax  ■  b  -  z  has  a 


solution  x  e  X.  By  taking  U  =  b  +  N,  we  deduce  the  desired  characteriza¬ 
tion.  To  show  the  last  statement,  suppose  that  d(y)  has  bounded  level  sets 

T 

but  there  exists  a  nonzero  vector  u  such  that  u  A  =  0.  Choose  a  vector 
T 

v  so  that  u  v  ^  0  and  b  ±  v  e  U.  Then  there  exist  vectors  x  and  x'  in 
in  X  with 

Ax  =  b  +  v  and  Ax'  =  b  -  v 
T 

Pre-multiplying  the  two  expressions  by  u  and  subtracting,  we  deduce 
T 

u  v  =  0  which  is  a  contradiction.  Therefore,  the  matrix  A  must  have 
linearly  independent  rows. 

From  the  above  Proposition,  we  see  that  the  boundedness  of  the  level  sets 

of  the  dual  function  d(y)  implies  a  kind  of  stability  property  on  the 

constraint  system  of  the  program  (20).  The  following  theorem  shows  that 

regardless  of  whether  this  stability  property  holds,  a  sequence  of  primal 
~  k  k 

vectors  {x(y  )}  induced  by  a  sequence  {y  }  which  is  generated  by  a  broad 

class  of  gradient-type  methods  applied  to  the  dual  program  (21)  will  always 

converge  to  unique  global  minimizer  of  (20),  provided  that  the  minimizer 

exists  and  that  X  is  either  a  polyhedral  cone  or  a  bounded  polyhedral  set. 

Theorem  6.1.  Let  D  be  a  symmetric  positive  definite  matrix  and  let  X  be 

either  a  polyhedral  cone  or  a  bounded  polyhedral  set.  Let  {y  }  be  a 

sequence  of  dual  variables  which  induces  a  sequence  of  primal  variables 
~  k  , 

(x(y  )}  where 

~  It  t  t 

x(y)  =  arg  min  -rxDx  +  cx  +  y  (b-Ax)  . 

xex 

Then  the  following  two  statements  are  equivalent: 

k  ,  * 

(K)  The  sequence  lx(y  )}  converges  to  the  unique  global  minimizer  x  of 

(20); 

(L)  The  sequence  {Vd(y  )}  converges  to  zero. 


i 


Theorem  6.1  has  captured  the  essence  of  the  convergence  results  in 
Corollaries  4.1  and  4.2  and  Theorem  5.1  by  giving  a  necessary  and  sufficient 
condition  on  the  dual  sequence  C y  }  in  order  for  the  corresponding  primal 
sequence  {x(y  )}  to  converge  to  the  unique  solution  of  (20).  To  prove 
Theorem  6.1,  we  need  the  following  lemma  which  summarizes  several  important 
properties  of  x(y)  considered  as  a  function  of  y. 

Lemma  6.1.  Let  D  be  a  symmetric  positive  matrix  of  order  n  and  let  X  be 

a  nonempty  convex  polyhedral  set  in  Rn.  Define  x  :  Rn  ♦  Rn  by 

*  1  T  T 

x  (q)  =  arg  min  —  x  Dx  +  q  x  . 

xex 


Then 

* 

(i)  x  (•)  is  a  well-defined  Lipschitz  continuous  function  of  q,  and 

there  exists  a  positive  constant  K  such  that 

*1  *22  12T*1*5  12 

lx  (q  )  -  x  (q  )l  <  -K(q'-q  jT(x  (q')-x  (q2))  for  all  q',q2  .  (23) 

In  particular,  if  {x  (q  )}  and  {x  (p  )}  are  two  sequences  such  that 

q*  -  pk  +  0,  then  x  (q*)  -  x*(p*)  *  0. 

(ii)  If  Q  is  convex  polyhedral  set,  then  the  image  x  (Q)  is  closed.  In 

*  k  k 

particular,  if  (x  (q  )}  is  a  converging  sequence  with  {q  }  Q,  then  there 

*  *  h  *  “ 

exists  a  vector  q  e  Q  such  that  X  (q  )  ♦  x  (q). 

(iii)  If  X  is  a  polyhedral  cone,  then 

*  * 

x  (aq)  <*  ax  (q)  for  all  a  >  0  .  (24) 

Proof .  (i)  The  well-definedness  of  x  (•)  is  clear.  To  show  (23),  we  note 

that 

(x-x*(q) )T(Dx*(q)+q)  1  0  for  all  x  6  X  (25) 

by  the  variational  inequality  characterization  of  x  (q).  Thus,  it  follows 


that 

(x*(q1)-x*(q2))T(Dx*(q2)+q2)  l  0  and 
Adding  the  two  inequalities  gives 


(x*(q2)-x*(q1))T(Dx*(q1)+q1)  l  0  . 
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*1  *2T  *1  *2  1  2  *P  *  1  *2 

(x  (q')-x  (q^))rD(x  (q  )-x  (q^))  £  -(q’-qZ)r(x  (q1)-x  (q^))  . 

By  taking  K  to  be  the  reciprocal  of  the  smallest  eigenvalue  of  D,  we 

obtain  (23).  From  (23),  we  deduce 

lx  (qS-x  ( q^  ) I  j  <  Klq^-q^l  j  for  all  qVq2 

* 

which  establishes  the  Lipschitz  continuity  of  x  (q)  as  well  as  the  last 
conclusion  of  (i). 

* 

(ii)  If  Q  is  a  convex  polyhedral  set,  then  x  (Q)  is  the  union  of  a 
finite  number  of  polyhedral  sets  (see  Ref.  17)  and  is  therefore  closed.  The 
assertion  about  the  sequence  {x  (q  )}  is  immediate. 

(iii)  Obviously,  (24)  holds  for  a  =  0.  Let  a  >  0.  Multiplying  (25)  by  the 

2 

scalar  a  ,  we  obtain 

*  w  * 

(ax  -  ax  (q))  (D(ax  (q) )  +  aq)  >  0  for  all  x  e  X  . 

* 

Since  X  is  a  cone,  aX  =  X,  thus  by  the  uniqueness  of  x  (aq),  we  conclude 
that  (24)  holds. 

Proof  of  Theorem  6.1.  (K)  ==>  (L).  By  (22),  we  have 

Vd( yk )  =  b  -  Ax (yk )  b  -  Ax  *  0 

* 

because  x  solves  (20). 

(L)  ==>  (K).  We  first  prove  that  the  sequence  {x(y  )}  is  bounded.  This  is 
certainly  true  if  the  set  X  is  bounded.  So,  let  X  be  a  polyhedral  cone. 

>v 

Suppose  that  {x(y  )}  is  unbounded.  Then  lx(y  )l  *■<*>.  The  normalized 

sequence  {x(y  )/lx(y  )!}  has  an  accumulation  point  which  must  be  nonzero. 

Let  u  be  one  such  point  and  with  no  loss  of  generality,  we  may  assume  that 
~  k  ~  k  k  ~  k 

x(y  )/lx(y  )l  +  u.  Since  Vd(y  )  =  b  -  Ax(y  )  ♦  0,  we  deduce  Au  *  0.  By 
Lemma  6.1  (iii),  we  may  write 


~  -  XVr*-  -  x*((c-ATyk)/lx(yk).) 
Ix(y  )l  Sx(y  )l 


^  T  k  ^  k  n  n 

By  Lemma  6.1(1),  the  sequence  {x  (-A  y  /lx(y  )l)  +  u.  Since  A  (R  )  is  a 

A 

convex  polyhedral  set.  Lemma  6.1(ii)  implies  the  existence  of  a  vector  y 

*  TA  *  TA  1  TA 

such  that  u  ■  x  (Ay).  Hence  Ax  (A  y)  *  0.  By  letting  q  *  A  y  and 

2  *  T 

q  =  0,  we  deduce  from  (23)  that  u  *  x  (A  y)  =0  which  is  a  contradiction. 

~  k  A 

Consequently,  the  sequence  {x(y  )}  is  bounded.  Let  x  be  any  accumulation 

point  of  {x(y  )}.  By  Lemma  6.1(ii)  again,  there  exists  a  vector  y  such 

A  ~  A  A  k  ~  k 

that  x  =  x(y).  Obviously  x  e  X.  Since  Vd(y  )  =  b  -  Ax(y  )  ♦  0,  it 

A  A 

follows  that  b  -  Ax  *  0.  Thus  the  vector  x  is  feasible  to  (20).  Moreover, 
we  have 

—  xDx  +  c  x  *  —  xDx  +  c  x  +  y(b-Ax)  *  d(y) 

A  it  T  AT 

because  x  *  x(y)  *  arg  min  “xDx+cx+y  (b-Ax).  Therefore  x  solves  the 

xex  2 

primal  program  (20).  But  since  (20)  has  a  unique  minimizer,  say  x  ,  the 

* 

sequence  {x(y  )}  must  converge  to  x  .  This  completes  the  proof  of  the 
theorem . 

We  should  point  out  that  condition  (L)  holds  for  a  large  class  of 
gradient-type  ascent  methods  which  include  many  be at -known  quasi-Newton 
methods  with  suitably  chosen  steplength  rules,  see  Refs.  18  and  19  e.g. 
According  to  Theorem  6.1,  these  gradient  methods  will  all  produce  a  primal 
sequence  converging  to  the  unique  solution  of  the  original  program  (20). 
Typically,  the  primal  sequence  {x(y  )}  is  generated  while  the  dual  sequence 
{y  }  is  being  computed. 

Basically,  the  proof  of  Theorem  6.1  shows  that  assertion  (L)  «=»>  (K) 
requires  no  particular  assumption  on  the  set  X.  On  the  other  hand,  the 
reverse  implication  (K)  **>  (L)  requires  that  X  be  either  a  cone  or 
bounded.  At  present  time,  we  cannot  prove  (K)  =»»>  (L)  without  such 
assumption  on  X.  We  leave  this  as  an  open  conjecture  and  encourage  the 
readers  to  prove  or  disprove  it. 


I 
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